Performance tuning for serving frameworks often feels like a "black box," making issues difficult to pinpoint (for example, slower responses as requests increase, performance differences in different devices).
msServiceProfiler provides end-to-end performance profiling. It clearly displays the performance of framework scheduling and model inference, helping users quickly locate performance bottlenecks and effectively improve performance.
[object Object]
- Before using msServiceProfiler, read about the restrictions in "" in the .
- Ensure that the corresponding service framework has been installed and its availability has been verified (the service starts successfully, and can process an inference request using official example scripts or APIs).
- MindIE Motor: Install and configure MindIE as described in and ensure that the MindIE Motor service can start successfully and complete a sample inference request.
- vLLM-ascend: Set up the environment and verify that the vLLM service can provide inference APIs for external systems. For details, see and official vLLM-ascend installation document.
- SGLang: Set up the environment and verify that the SGLang service can provide inference APIs for external systems. For details, see and official SGLang installation document.
[object Object]
To enable msServiceProfiler's profiling capability, set the environment variable [object Object] before service deployment. If the environment variable is misspelled or is not set before service deployment, the profiling capability cannot be enabled.
General Note
[object Object]: specifies the path to the performance profiling configuration file (JSON). This file controls whether profiling is enabled, specifies the data output directory, and configures other settings.
Example (using the configuration file in the current working directory as an example)
[object Object]
The value of [object Object] must include the JSON file name. The JSON file is the configuration file for controlling profile data collection. For example, it specifies the path for storing profile metadata and enables or disables operator collection. For details about the fields, see . If no configuration file exists at the specified path, the tool automatically generates a default configuration (with the profiling feature disabled by default).
[object Object]
The steps for starting a service vary by framework. However, for msServiceProfiler, the key point is to configure the environment variables before the service process starts. After that, start the service using the standard method for each respective framework.
Start the inference service as described in the MindIE Installation Guide. If [object Object] is correctly configured, logs prefixed with [object Object] will be output before the service deployment completes, indicating that msServiceProfiler has been initialized. Example:
If the configuration file specified by [object Object] does not exist, the tool automatically creates a configuration file. The log is similar to the following:
After preparing the vLLM-ascend environment and configuring the necessary variables, start the service using the native vLLM method. Example:
For first-time integration, you need to integrate msServiceProfiler into the SGLang serving startup entry point. After that, start the service as usual.
In the preceding command,
[object Object]: specifies the path to the profiling configuration file. If the file does not exist, it is automatically generated.
Once the service is successfully deployed, you can precisely control the profiling behavior by modifying the fields in the configuration file specified by [object Object]. (Only three fields are used as examples here):
Table 1 Parameters
Generally, if [object Object] remains [object Object], the tool collects data from the moment the service receives a request until the request ends. The size of the directory under [object Object] will continue to grow. Therefore, it is advised to collect data only during critical time periods.
Whenever the [object Object] field changes, the tool outputs corresponding logs to indicate the change.
When [object Object] is changed from [object Object] to [object Object], all fields in the configuration file are dynamically reloaded by the tool.
Similarly, the tool generates the original profile data of the inference service in [object Object] (default: [object Object]).
Install environment dependencies.
[object Object]Run the parsing command (general form):
[object Object][object Object]is set to the path specified by[object Object]in . After parsing, parsed profile data files are generated in the directory where the command is executed.
[object Object]
The parsed profile data is available in [object Object], [object Object], and [object Object] formats. You can perform quick analysis from different dimensions (such as requests and scheduling) using the CSV files, or import the [object Object] or [object Object] files into MindStudio Insight for visualization. For detailed instructions and analysis explanations, see the section "Serving Tuning" in the .
The following figure shows an example of profile data visualized using MindStudio Insight.